perm filename WEIZEN.REV[PUB,JMC]2 blob
sn#211825 filedate 1976-04-24 generic text, type C, neo UTF8
COMMENT ⊗ VALID 00008 PAGES
C REC PAGE DESCRIPTION
C00001 00001
C00002 00002 .require "memo.pub[let,jmc]" source
C00013 00003
C00020 00004
C00026 00005 .item←0
C00033 00006
C00056 00007
C00066 00008
C00079 ENDMK
C⊗;
.require "memo.pub[let,jmc]" source;
.cb AN UNREASONABLE BOOK
.skip 2
.begin indent 0;
Joseph Weizenbaum, %2Computer Power and Human Reason%1, W. H. Freeman Co.,
San Francisco 1975
.end;
This moralistic and incoherent book uses computer science and
technology as an illustration to support the view promoted by Lewis
Mumford, Theodore Roszak, and Jacques Ellul, that science has led to
an immoral view of man and the world.
I am frightened by its arguments that certain research should not be
done if it is based on or might result in an "obscene" picture of the
world and man. Worse yet, the book's notion of "obscenity" is vague
enough to admit arbitrary interpretations by activist bureaucrats.
.cb IT'S HARD TO FIGURE OUT WHAT HE REALLY BELIEVES ...
Weizenbaum's style involves making extreme statements which
are later qualified by contradictory statements. Therefore, almost
any quotation is out of context, making it difficult to
summarize his contentions accurately.
The following passages illustrate the difficulty:
%2"In 1935, Michael Polanyi"%1, [British chemist and
philosopher of science, was told by] %2"Nicolai Bukharin, one of the
leading theoreticians of the Russian Communist party, ... [that]
'under socialism the conception of science pursued for its own sake
would disappear, for the interests of scientists would spontaneously
turn to the problems of the current Five Year Plan.' Polanyi sensed
then that 'the scientific outlook appeared to have produced a
mechanical conception of man and history in which there was no place
for science itself.' And further that 'this conception denied
altogether any intrinsic power to thought and thus denied any grounds
for claiming freedom of thought.'"%1 - from page 1. Well, that's
clear enough; Weizenbaum favors freedom of thought and science and is
worried about threats to them. But on page 265, we have
%2"Scientists who continue to prattle on about 'knowledge for its own
sake' in order to exploit that slogan for their self-serving ends
have detached science and knowledge from any contact with the real
world".%1 Here Weizenbaum seems to be against pure science, i.e.
research motivated solely by curiosity. We also have
%2"With few exceptions, there have been no results, from over twenty
years of artificial intelligence research, that have found their way
into industry generally or into the computer industry in
particular"%1. - page 229 This again suggests that industrial
results are necessary to validate science.
%2"Science promised man power. But as so often happens when people
are seduced by promises of power ... the price actually paid is
servitude and impotence"%1. This is from the book jacket. Presumably
the publisher regards it as a good summary of the book's main point.
%2"I will, in what follows, try to maintain the position that there
is nothing wrong with viewing man as an information processor (or
indeed as anything else) nor with attempting to understand him from
that perspective, providing, however, that we never act as though
any single perspective can comprehend the whole man."%1 - page 140.
We can certainly live with that, but
%2"Not only has our unbounded feeding on science caused us to become
dependent on it, but, as happens with many other drugs taken in
increasing dosages, science has been gradually converted into a slow
acting poison".%1 - page 13. These are qualified by
%2"I argue for the rational use of science and technology, not for
its mystification, let alone its abandonment"%1. - page 256
In reference to the proposal for a moratorium on certain experiments
with recombinant DNA because they might be dangerous, we
have %2"Theirs is certainly a step in the right direction, and their
initiative is to be applauded. Still, one may ask, why do they feel
they have to give a reason for what they recommend at all? Is not
the overriding obligation on men, including men of science, to exempt
life itself from the madness of treating everything as an object, a
sufficient reason, and one that does not even have to be spoken? Why
does it have to be explained? It would appear that even the noblest
acts of the most well-meaning people are poisoned by the corrosive
climate of values of our time."%1 Is Weizenbaum against all
experimental biology or even all experiments with DNA? I would
hesitate to conclude so from this quote; he may say the direct
opposite somewhere else.
%2"Those who know who and what they are do not need to ask what they
should do."%1 - page 273. Let me assure the reader that there is
nothing in the book that offers any way to interpret this
pomposity. The menace of such grandiloquent precepts is that they
require a priesthood to apply them to particular cases, and would-be
priests quickly crystallize around any potential center of power.
A corollary of this is that people can be attacked for what they are
rather than for anything specific they have done. The April 1976
issue of ⊗Ms. has a poignant illustration of this in an article about
"trashing".
%2"An individual is dehumanized whenever he is treated as less than a
whole person"%1. - page 266. This is also subject to priestly
interpretation as in the encounter group movement.
%2"The first kind %1[of computer application]%2 I would call simply
obscene. These are ones whose very contemplation ought to give rise
to feelings of disgust in every civilized person. The proposal I
have mentioned, that an animal's visual system and brain be coupled
to computers, is an example. It represents an attack on life itself.
One must wonder what must have happened to the proposers' perception
of life, hence to their perceptions of themselves as part of the
continuum of life, that they can even think of such a thing, let alone
advocated it"%1. No argument is offered that might be answered,
and no attempt is made to define criteria of acceptability. I think
Weizenbaum and the scientists who have praised the book may be
surprised at some of the repressive uses to which the book will be
put. However, they will be able to point to passages in the book
with quite contrary sentiments, so the repression won't be their fault.
.CB BUT HERE'S A TRY AT SUMMARIZING:
.item←0;
As these inconsistent passages show, it isn't easy to
determine Weizenbaum's position, but the following seem to be the
book's main points:
.bb "#. Computers cannot be made to reason usefully about human affairs."
This is supported by quoting overoptimistic predictions by computer
scientists and giving examples of non-verbal human communication.
However, Weizenbaum doesn't name any specific task that computers
cannot carry out, because he wishes %2"to avoid the unnecessary,
interminable, and ultimately sterile exercise of making a catalogue
of what computers will and will not be able to do, either here and
now or ever"%1. It is also stated that human and machine reasoning
are incomparable and that the sensory experience of a human is
essential for human reasoning.
.bb "#. There are tasks that computers should not be programmed to do."
Some are tasks Weizenbaum thinks shouldn't be done at all -
mostly for new left reasons. One may quarrel with his politics, and
I do, but obviously computers shouldn't do what shouldn't be done.
However, Weizenbaum also objects to computer hookups to animal brains
and computer conducted psychiatric interviews. As to the former, I
couldn't tell whether he is an anti-vivisectionist, but he seems to
have additional reasons for calling them "obscene". The objection to
computers doing psychiatric interviews also has a component beyond
the conviction that they would necessarily do it badly. Thus he
says, %2"What can the psychiatrist's image of his patient be when he
sees himself, as a therapist, not as an engaged human being acting as
a healer, but as an information processor following rules, etc.?"%1
This seems like the renaissance era religious objections to
dissecting the human body that came up when science revived. Even
the Popes eventually convinced themselves that regarding the body as
a machine for scientific or medical purposes was quite compatible
with regarding it as the temple of the soul. Recently they have
taken the same view of studying mental mechanisms for scientific or
psychiatric purposes.
.bb "#. Science has led people to a wrong view of the world and of life."
The view is characterized as mechanistic, and the example of
clockwork is given. (It seems strange for a computer scientist to
give this example, because the advance of the computer model over
older mechanistic models is that computers can and clockwork can't
make decisions.) Apparently analysis of a living system as composed
of interacting parts rather than treating it as an unanalyzed whole
is bad.
.bb "#. Science is not the sole or even main source of reliable general knowledge."
However, he doesn't
propose any other sources of knowledge or say what the limits of
scientific knowledge is except to characterize certain thoughts as
"obscene".
.bb "#. Certain people and institutions are attacked."
These include the
Department of "Defense" (sic), %2Psychology Today%1, the %2New York
Times%1 Data Bank, compulsive computer programmers, Kenneth Colby,
Marvin Minsky, Roger Schank, Allen Newell, Herbert Simon, J.W.
Forrester, Edward Fredkin, B.F. Skinner, Warren McCulloch (until he
was old), Laplace and Leibniz.
.bb "#. Certain political and social views are taken for granted."
The view that U.S. policy in Vietnam was "murderous" is used
to support an attack on "logicality" (as opposed to "rationality")
and the view of science as a "slow acting poison". The phrase
%2"It may be that the people's cultivated and finally addictive
hunger for private automobiles..."%1 (p.30) makes psychological,
sociological, political, and technological presumptions all in
one phrase. Similarly, %2"Men could instead choose to have truly
safe automobiles, decent television, decent housing for everyone,
or comfortable, safe, and widely distributed mass transportation."%1
presumes wide agreement about what these things are, what is
technologically feasible, what the effects of changed policies would
be, and what activities aimed at changing people's taste are
permissible for governments.
.cb THE ELIZA EXAMPLE
Perhaps the most interesting part of the book is the account
of his own program ELIZA that parodies Rogerian non-directive
psychotherapy and his anecdotal account of how some people ascribe
intelligence and personality to it. In my opinion, it is quite
natural for people who don't understand the notion of algorithm to
imagine that a computer computes analogously to the way a human
reasons. This leads to the idea that accurate computation entails
correct reasoning and even to the idea that computer malfunctions are
analogous to human neuroses and psychoses. Actually, programming a
computer to draw interesting conclusions from premises is very
difficult and only limited success has been attained. However, the
effect of these natural misconceptions shouldn't be exaggerated;
people readily understand the truth when it is explained, especially
when it applies to a matter that concerns them. In particular, when
an executive excuses a mistake by saying that he placed excessive
faith in a computer, a certain skepticism is called for.
Colby's (1973) study is interesting in this connection, but
the interpretation below is mine. Colby had psychiatrists interview
patients over a teletype line and also had them interview his PARRY
program that simulates a paranoid. Other psychiatrists were asked to
decide from the transcripts whether the interview was with a man or
with a program, and they did no better than chance. However, since
PARRY is incapable of the simplest causal reasoning, if you ask, "How
do you know the people following you are Mafia" and get a reply that
they look like Italians, this must be a man not PARRY. Curiously, it
is easier to imitate (well enough to fool a psychiatrist) the
emotional side of a man than his intellectual side. Probably the
subjects expected the machine to have more logical ability, and this
expectation contributed to their mistakes. Alas, random selection
from the directory of the Association for Computing Machinery did no
better.
It seems to me that ELIZA and PARRY show only that people,
including psychiatrists, often have to draw conclusions on slight
evidence, and are therefore easily fooled. If I am right, two
sentences of instruction would allow them to do better.
In his 1966 paper on ELIZA (cited as 1965),
Weizenbaum writes,
%2"One goal for an augmented ELIZA program is thus a system which
already has access to a store of information about some aspect of the
real world and which, by means of conversational interaction with
people, can reveal both what it knows, i.e. behave as an information
retrieval system, and where its knowledge ends and needs to be
augmented. Hopefully the augmentation of its knowledge will also be
a direct consequence of its conversational experience. It is
precisely the prospect that such a program will converse with many
people and learn something from each of them which leads to the hope
that it will prove an interesting and even useful conversational
partner."%1 Too bad he didn't successfully pursue this goal; no-one
else has. I think success would have required a better
understanding of formalization than is exhibited in the book.
.item←0;
.CB WHAT DOES HE SAY ABOUT COMPUTERS?
While Weizenbaum's main conclusions concern science in
general and are moralistic in character, some of his remarks about
computer science and AI are worthy of comment.
#. He concludes that since a computer cannot have the
experience of a man, it cannot understand a man. There are three
points to be made in reply. First, humans share each other's
experiences and those of machines or animals only to a limited
extent. In particular, men and women have different experiences.
Nevertheless, it is common in literature for a good writer to
show greater understanding of the experience of the opposite sex
than a poorer writer of that sex. Second, the notion of experience is
poorly understood; if we understood it better, we could reason about
whether a machine could have a simulated or vicarious experience
normally confined to humans. Third, what we mean by understanding is
poorly understood, so we don't yet know how to define whether a
machine understands something or not.
#. Like his predecessor critics of artificial intelligence,
Taube, Dreyfus and Lighthill,
Weizenbaum is impatient, implying that if the problem hasn't been
solved in twenty years, it is time to give up. Genetics took about a
century to go from Mendel to the genetic code for proteins, and still
has a long way to go before we will fully understand the genetics
and evolution of intelligence and behavior. Artificial intelligence
may be just as difficult. My current answer to the question of when
machines will reach human-level intelligence is that a precise
calculation shows that we are between 1.7 and 3.1 Einsteins and .3
Manhattan Projects away from the goal. However, the current research
is producing the information on which the Einstein will base himself
and is producing useful capabilities all the time.
#. The book confuses computer simulation of a phenomenon with
its formalization in logic. A simulation is only one kind of
formalization and not often the most useful - even to a computer. In
the first place, logical and mathematical formalizations can use
partial information about a system insufficient for a simulation.
Thus the law of conservation of energy tells us much about possible
energy conversion systems before we define even one of them. Even
when a simulation program is available, other formalizations are
necessary even to make good use of the simulation. This review isn't
the place for a full explanation of the relations between these
concepts.
Like %2Punch%1's famous curate's egg, the book is good in
parts. Thus it raises the following interesting issues:
.item←0;
#. What would it mean for a computer to hope or be desperate
for love? Answers to these questions depend on being able to
formalize (not simulate) the phenomena in question. My guess is that
adding a notion of hope to an axiomatization of belief and wanting
might not be difficult. The study of %2propositional attitudes%1 in
philosophical logic points in that direction.
#. Do differences in experience make human and machine
intelligence necessarily so different that it is meaningless to ask
whether a machine can be more intelligent than a machine? My opinion
is that comparison will turn out to be meaningful. After all, most
people have not doubt that humans are more intelligent than turkeys.
Weizenbaum's examples of the dependence of human intelligence on
sensory abilities seem even refutable, because we recognize no
fundamental difference in humanness in people who are severely
handicapped sensorily, e.g. the deaf, dumb and blind or paraplegics.
.cb IN DEFENSE OF THE UNJUSTLY ATTACKED - SOME OF WHOM ARE INNOCENT
Here are defenses of Weizenbaum's targets. They are not
guaranteed to entirely suit the defendees.
Weizenbaum's conjecture that the Defense Department supports
speech recognition research in order to be able to snoop on telephone
conversations is biased, baseless, false, and seems motivated by
political malice. The committee of scientists that proposed the
project advanced quite different considerations, and the high
officials who made the final decisions are not ogres. Anyway their
other responsibilities leave them no time for complicated and devious
considerations. I put this one first, because I think the failure of
many scientists to defend the Defense Department against attacks they
know are unjustified, is unjust in itself, and furthermore has harmed
the country.
Weizenbaum doubts that computer speech recognition will have
cost-effective applications beyond snooping on phone conversations.
He also says, %2"There is no question in my mind that there is no
pressing human problem that will be more easily solved because such
machines exist"%1. I worry more about whether the programs can be
made to work before the sponsor loses patience. Once they work,
costs will come down. Winograd pointed out to me that many possible
household applications of computers may not be feasible without some
computer speech recognition. One needs to think %3both%1 about how to
solve recognized problems %3and%1 about opportunities to put new
technological possibilities to good use. The telephone was not
invented by a committee considering already identified problems of
communication.
Referring to %2Psychology Today%1 as a cafeteria simply
excites the snobbery of those who would like to consider their
psychological knowledge to be above the popular level. So far as I
know, professional and academic psychologists welcome the opportunity
offered by %2Psychology Today%1 to explain their ideas to a wide
public. They might even buy a cut-down version of Weizenbaum's book
if he asks them nicely. Hmm, they might even buy this review.
Weizenbaum has invented a %2New York Times Data Bank%1
different from the one operated by the %2New York Times%1 - and
possibly better. The real one stores abstracts written by humans and
doesn't use the tapes intended for typesetting machines. As a result
the user has access only to abstracts and cannot search on features
of the stories themselves, i.e. he is at the mercy of what the
abstractors thought was important at the time.
Using computer programs as psychotherapists, as Colby
proposed, would be moral if it would cure people. Unfortunately,
computer science isn't up to it, and maybe the psychiatrists aren't
either.
I agree with Minsky in criticizing the reluctance of art
theorists to develop formal theories. George Birkhoff's formal
theory was probably wrong, but he shouldn't have been criticized for
trying. The problem seems very difficult to me, and I have made no
significant progress in responding to a challenge from Arthur
Koestler to tell how a computer program might make or even recognize
jokes. Perhaps some reader of this review might have more success.
There is a whole chapter attacking "compulsive computer
programmers" or "hackers". This mythical beast lives in the computer
laboratory, is an expert on all the ins and outs of the time-sharing
system, elaborates the time-sharing system with arcane features that
he never documents, and is always changing the system before he even
fixes the bugs in the previous version. All these vices exist, but I
can't think of any individual who combines them, and people generally
outgrow them. As a laboratory director, I have to protect the
interests of people who program only part time against tendencies to
over-complicate the facilities. People who spend all their time
programming and who exchange information by word of mouth sometimes
have to be pressed to make proper writeups. The other side of the
issue is that we professors of computer science sometimes lose our
ability to write actual computer programs through lack of practice
and envy younger people who can spend full time in the laboratory.
The phenomenon is well known in other sciences and in other human
activities.
Weizenbaum attacks the Yale computer linguist, Roger Schank,
as follows - the inner quotes are from Schank: %2"What is contributed when it is
asserted that 'there exists a conceptual base that is interlingual,
onto which linguistic structures in a given language map during the
understanding process and out of which such structures are created
during generation [of linguistic utterances]'? Nothing at all. For
the term 'conceptual base' could perfectly well be replaced by the
word 'something'. And who could argue with that so-transformed
statement?"%1 Weizenbaum goes on to say that the real scientific
problem "remains as untouched as ever". On the next page he says
that unless the "Schank-like scheme" understood the sentence %2"Will
you come to dinner with me this evening?"%1 to mean %2"a shy young
man's desperate longing for love%1, then the sense in which the
system "understands" is "about as weak as the sense in which ELIZA
"understood"". This good example raises interesting issues and seems
to call for some distinctions. Full understanding of the sentence
indeed results in knowing about the young man's desire for love, but
it would seem that there is a useful lesser level of understanding in
which the machine would know only that he would like her to come to
dinner.
Contrast Weizenbaum's demanding,
more-human-than-thou attitude to Schank and Winograd
with his respectful and even obsequious attitude to Chomsky. We have
%2"The linguist's first task is therefore to write grammars, that is,
sets of rules, of particular languages, grammars capable of
characterizing all and only the grammatically admissible sentences of
those languages, and then to postulate principles from which crucial
features of all such grammars can be deduced. That set of principles
would then constitute a universal grammar. Chomsky's hypothesis is,
to put it another way, that the rules of such a universal grammar
would constitute a kind of projective description of important
aspects of the human mind."%1 There is nothing here demanding
that the universal grammar take into account the young man's desire
for love. As far as I can see, Chomsky is just as much a rationalist
as we artificial intelligentsia.
Chomsky's goal of a universal grammar and Schank's goal of a
conceptual base are similar, except that Schank's ideas are further
developed, and the performance of his students' programs can be
compared with reality. I think they will require drastic revision
and may not be on the right track at all, but then I am pursuing a
rather different line of research concerning how to represent the
basic facts that an intelligent being must know about the world. My
idea is to start from epistemology rather than from language,
regarding their linguistic representation as secondary. This approach
has proved difficult, has attracted few practitioners, and has led to
few computer programs, but I still think it's right.
Weizenbaum approves of the Chomsky school's haughty attitude
towards Schank, Winograd and other AI based language researchers. On
page 184, he states, %2"many linguists, for example, Noam Chomsky,
believe that enough thinking about language remains to be done to
occupy them usefully for yet a little while, and that any effort to
convert their present theories into computer models would, if
attempted by the people best qualified, be a diversion from the main
task. And they rightly see no point to spending any of their
energies studying the work of the hackers."%1
This brings the chapter on "compulsive computer programmers"
alias "hackers" into a sharper focus. Chomsky's latest book
%2Reflections on Language%1 makes no reference to the work of
Winograd, Schank, Charniak, Wilks, Bobrow or William Woods to name
only a few of those who have developed large computer systems that
work with natural language and who write papers on the semantics of
natural language. The actual young computer programmers who call
themselves hackers and who come closest to meeting Weizenbaum's
description don't write papers on natural language. So it seems
that the hackers whose work need not be studied are Winograd, Schank,
et. al. who are professors and senior scientists. The Chomsky
school may be embarassed by the fact that it has only recently
arrived at the conclusion that the semantics of natural language is
more fundamental than its syntax, while AI based researchers have
been pursuing this line for fifteen years.
The outside observer should be aware that to some extent this
is a pillow fight within M.I.T. Chomsky and Halle are not to be
dislodged from M.I.T. and neither is Minsky - whose students have
pioneered the AI approach to natural language. Schank is quite
secure at Yale. Weizenbaum also has tenure. However, some assistant
professorships in linguistics may be at stake, especially at M.I.T.
Allen Newell and Herbert Simon are criticized for being
overoptimistic and are considered morally defective for attempting to
describe humans as difference-reducing machines. Simon's view that
the human is a simple system in a complex environment is singled out
for attack. In my opinion, they were overoptimistic, because their
GPS model on which they put their bets wasn't good enough. Maybe
Newell's current %2production system models%1 will work out better.
As to whether human mental structure will eventually turn out to be
simple, I vacillate but incline to the view that it will turn out to
be one of the most complex biological phenomena.
I regard Forrester's models as incapable of taking into
account qualitative changes, and the world models they have built as
defective even in their own terms, because they leave out
saturation-of-demand effects that cannot be discovered by curve-fitting
as long as a sytem is rate-of-expansion limited. Moreover, I don't
accept his claim that his models are better suited than the unaided
mind in "interpreting how social systems behave", but Weizenbaum's
sarcasm on page 246 is unconvincing. He quotes Forrester,
"[desirable modes of behavior of the social system] %2seem to be
possible only if we have a good understanding of the system dynamics
and are willing to endure the self-discipline and pressures that must
accompany the desirable mode"%1. Weizenbaum comments, %2"There is
undoubtedly some interpretation of the words 'system' and 'dynamics'
which would lend a benign meaning to this observation"%1. Sorry, but
it looks ok to me provided one is suitably critical of Forrester's
proposed social goals and the possibility of making the necessary
assumptions and putting them into his models.
Skinner's behaviorism that refuses to assign reality to
people's internal state seems wrong to me, but we can't call him
immoral for trying to convince us of what he thinks is true.
Weizenbaum quotes Edward Fredkin, former director of Project
MAC, and the late Warren McCulloch of M.I.T. without giving their
names. pp. 241 and 240. Perhaps he thinks a few puzzles will make
the book more interesting, and this is so. Fredkin's plea for
research in automatic programming seems to overestimate the extent to
which our society currently relies on computers for decisions. It
also overestimates the ability of the faculty of a particular
university to control the uses to which technology will be put, and
it underestimates the difficulty of making knowledge based systems of
practical use. Weizenbaum is correct in pointing out that Fredkin
doesn't mention the existence of genuine conflicts in society, but
only the new left sloganeering elsewhere in the book gives a hint as
to what he thinks they are and how he proposes to resolve them.
As for the quotation from (McCulloch 1956), Minsky tells me
"this is a brave attempt to find a dignified sense of freedom within
the psychological determinism morass". Probably this can be done
better now, but Weizenbaum wrongly implies that McCulloch's 1956
effort is to his moral discredit.
Finally, Weizenbaum attributes to me two statements - both
from oral presentations - which I cannot verify. One of them is
%2"The only reason we have not yet succeeded in simulating every
aspect of the real world is that we have been lacking a sufficiently
powerful logical calculus. I am working on that problem."%1 This
statement doesn't express my present opinion or my opinion in 1973
when I am alleged to have expressed it in a debate, and no-one has
been able to find it in the video-tape of the debate.
We can't simulate "every aspect of the real world", because
the initial state information is never available, the laws of motion
are imperfectly known, and the calculations for a simulation are too
extensive. Moreover, simulation wouldn't necessarily answer our
questions. Instead, we must find out how to represent in the memory
of a computer the information about the real world that is actually
available to a machine or organism with given sensory capability, and
also how to represent a means of drawing those useful conclusions
about the effects of courses of action that can be correctly inferred
from the attainable information. Having a %2sufficiently powerful
logical calculus%1 is an important part of this problem - but one of
the easier parts.
The second quotation from me is the rhetorical question, %2"What
do judges know that we cannot tell a computer"%1. I'll stand on that
if we make it "eventually tell" and especially if we require that it
be something that one human can reliably teach another.
.cb A SUMMARY OF POLEMICAL SINS
The speculative sections of the book contain numerous dubious
little theories, such as this one about the dehumanizing effect of
of the invention of the clock: %2"The clock had created literally a new
reality; and that is what I meant when I said earlier that the trick
man turned that prepared the scene for the rise of modern science was
nothing less than the transformation of nature and of his perception
of reality. It is important to realize that this newly created
reality was and remains an impoverished version of the older one, for
it rests on a rejection of those direct experiences that formed the
basis for, and indeed constituted the old reality. The feeling of
hunger was rejected as a stimulus for eating; instead one ate when an
abstract model had achieved a certain state, i.e. when the hand of a
clock pointed to certain marks on the clock's face (the
anthropomorphism here is highly significant too), and similarly for
signals for sleep and rising, and so on."%1
This idealization of primitive life is simply thoughtless.
Like modern man, primitive man ate when the food was ready, and
primitive man probably had to start preparing it even further in
advance. Like modern man, primitive man lived in families whose
members are no more likely to become hungry all at once than are the
members of a present family.
I get the feeling that in toppling this microtheory I am not
playing the game; the theory is intended only to provide an
atmosphere, and like the reader of a novel, I am supposed to suspend
disbelief. But the contention that science has driven us from a
psychological Garden of Eden depends heavily on such word pictures.
By the way, I recall from my last sabbatical at M.I.T. that
the %2feeling of hunger%1 is more often the %2direct social stimulus
for eating%1 for the "hackers" deplored in Chapter 4 than it could
have been for primitive man. Often on a crisp New England
night, even as the clock strikes three, I hear them call to one
another, messages flash on the screens, a flock of hackers
magically gathers, and the whole picturesque assembly rushes
chattering off to Chinatown.
.item←0;
I find the book substandard as a piece of polemical writing
in the following respects:
#. The author has failed to work out his own
positions on the issues he discusses. Making an extreme statement in
one place and a contradictory statement in another is no substitute
for trying to take all the factors into account and reach a
considered position. Unsuspicious readers can come away with a great
variety of views, and the book can be used to support contradictory
positions.
#. The computer linguists - Winograd, Schank, et. al. - are
denigrated as hackers and compulsive computer programmers by
innuendo.
#. One would like to know more precisely what biological and
psychological experiments and computer applications he finds
acceptable. Reviewers have already drawn a variety of conclusions on
this point.
#. The terms "authentic", "obscene", and "dehumanization are
used as clubs. This is what mathematicians call "proof by
intimidation".
#. The book encourages a snobbery that has no need to argue
for its point of view but merely utters code words, on hearing which
the audience is supposed applaud or hiss as the case may be. The
%2New Scientist%1 reviewer certainly salivates in most of the
intended places.
#. Finally, when moralizing is both vehement and vague, it
invites authoritarian abuse either by existing authority or by new
political movements. Imagine, if you can, that this book were the
bible of some bureaucracy, e.g. an Office of Technology Assessment,
that acquired power over the computing or scientific activities of a
university, state, or country. Suppose Weizenbaum's slogans were
combined with %2the bureaucratic ethic%1 that holds that any problem
can be solved by a law forbidding something and a bureaucracy of
eager young lawyers to enforce it. Postulate further a vague
%2Humane Research Act%1 and a "public interest" organization with
more eager young lawyers suing to get judges to legislate new
interpretations of the Act. One can see a laboratory needing more
lawyers than scientists and a Humane Research Administrator capable
of forbidding or requiring almost anything.
I see no evidence that Weizenbaum forsees his work being used
in this way; he doesn't use the phrase %2laissez innover%1 which is
the would-be science bureaucrat's analogue of the economist's
%2laissez faire%1, and he never uses the indefinite phrase "it should
be decided" which is a common expression of the bureaucratic ethic.
However, he has certainly given his fellow computer scientists at
least some reason to worry about potential tyranny.
Let me conclude this section with a quotation from Andrew D. White,
the first president of Cornell University, that seems applicable to
the present situation - not only in computer science, but also
in biology. - "%2In all modern history, interference with science in
the supposed interest of religion, no matter how conscientious such
interference may have been, has resulted in the direst evils both to
religion and to science, and invariably; and, on the other hand,
all untrammelled scientific investigation, no matter how dangerous
to religion some of its stages my have seemed for the time to be,
has invariably resulted in the highest good both of religion and
of science"%1. Substitute ⊗morality for ⊗religion and the parallel
is clear. Frankly, the feebleness of the reaction to attacks on
scientific freedom worries me more than the strength of the attacks.
.CB WHAT WORRIES ABOUT COMPUTERS ARE WARRANTED?
Grumbling about Weizenbaum's mistakes and moralizing
is not enough. Genuine worries prompted the book, and many
people share them. Here are the genuine concerns that I can
identify and the opinions of one computer scientist about their
resolution: What is the danger that the computer will lead to a false
model of man? What is the danger that computers will be misused?
Can human-level artificial intelligence be achieved? What, if any,
motivational characteristics will it have? Would the achievement of
artificial intelligence be good or bad for humanity?
.item←0;
.bb "#. Does the computer model lead to a false model of man."
Historically, the mechanistic model of the life and the world
followed animistic models in accordance with which, priests and
medicine men tried to correct malfunctions of the environment and man
by inducing spirits to behave better. Replacing them by mechanistic
models replaced shamanism by medicine. Roszak explicitly would like
to bring these models back, because he finds them more "human", but
he ignores the sad fact that they don't work, because the world isn't
constructed that way. The pre-computer mechanistic models of the
mind were, in my opinion, unsuccessful,but I think the psychologists
pursuing computational models of mental processes may eventually
develop a really beneficial psychiatry.
Philosophical and moral thinking hasn't yet found a model
of man that relates human beliefs and purposes to the physical
world in a plausible way. Some of the unsuccessful attempts have
been more mechanistic than others. Both mechanistic and non-mechanistic
models have led to great harm when made the basis of political
ideology, because they have allowed tortuous reasoning to justify
actions that simple human intuition regards as immoral. In my
opinion, the relation between beliefs, purposes and wants to the
physical world is a complicated but ultimately solvable problem.
Computer models can help solve it, and can provide criteria
that will enable us to reject false solutions. The latter is
more important for now, and
computer models are already hastening the decay of dialectical
materialism in the Soviet Union.
.bb "#. What is the danger that computers will be misused?"
Up to now, computers have been just another labor-saving technology.
I don't agree with Weizenbaum's acceptance of the claim that our
society would have been inundated by paper work without computers.
Without computers, people would work a little harder and get a little
less for their work. However, when home terminals become available,
social changes of the magnitude of those produced by the telephone
and automobile will occur. I have discussed them elsewhere, and I
think they will be good - as were the changes produced by the
automobile and the telephone. Tyranny comes from control of the
police coupled with a tyrannical ideology; data banks will be a minor
convenience. No dictatorship yet has been overthrown for lack of a
data bank.
One's estimate of whether technology will work out well in
the future is correlated with one's view of how it worked out in the
past. I think it has worked out well - e.g. cars were not a mistake
- and am optimistic about the future. I feel that much current
ideology is a combination of older anti-scientific and
anti-technological views with new developments in the political
technology of instigating and manipulating fears and guilt feelings.
.bb "#. What motivations will artificial intelligence have?"
It will have what motivations we choose to give it. Those
who finally create it should start by motivating it only to answer
questions and should have the sense to ask for full pictures of the
consequences of alternate actions rather than simply how to achieve a
fixed goal, ignoring possible side-effects.
Giving it human motivational structure with its
shifting goals sensitive to physical state would require a deliberate
effort beyond that required to make it behave intelligently.
.bb "#. Will artificial intelligence be good or bad?"
Here we are talking about machines with the same range of
intellectual abilities as are posessed by humans. However, the
science fiction vision of robots with almost precisely the ability of
a human is quite unlikely, because the next generation of computers
or even hooking computers together would produce an intelligence that
might be qualitatively like that of a human, but thousands of times
faster. What would it be like to be able to put a hundred years
thought into every decision? I think it is impossible to say whether
qualitatively better answers would be obtained; we will have to try
it and see.
The achievement of above-human-level artificial intelligence
will open to humanity an incredible variety of options. We cannot
now fully envisage what these options will be, but it seems apparent
that one of the first uses of high-level artificial intelligence will
be to determine the consequences of alternate policies governing its
use. I think the most likely variant is that man will use artificial
intelligence to transform himself, but once its properties and
the conequences of its use are known, we may decide not to use it.
Science would then be a sport like mountain climbing; the point would
be to discover the facts about the world using some stylized limited
means. I wouldn't like that, but once man is confronted by the
actuality of full AI, they may find our opinion as relevant to them
as we would find the opinion of ⊗Pithecanthropus about whether
subsequent evolution took the right course.
.bb "#. What shouldn't computers be programmed to do."
Obviously one shouldn't program computers to do things that
shouldn't be done. Moreover, we shouldn't use programs to mislead
ourselves or other people.
Apart from that, I find none of Weizenbaum's examples convincing.
However, I doubt the advisability of making robots with human-like
motivational and emotional structures that might have rights and
duties independently of humans. Moreover, I think it might be
dangerous to make a machine that evolved intelligence by responding
to a program of rewards and punishments unless its trainers
understand the intellectual and motivational structure being
evolved.
All these questions merit and have received more extensive
discussion, but I think the only rational policy now is to expect the
people confronted by the problem to understand their best interests
better than we now can. Even if full AI were to arrive next year,
this would be right. Correct decisions will require an intense
effort that cannot be mobilized to consider an eventuality that is
still remote. Imagine asking the presidential candidates to debate
on TV what each of them would do about each of the forms that full AI
might take.
.bb References:
McCulloch,W.S.(1956) Toward some circuitry of ethical robots or an
observational science of the genesis of social evaluation in the
mind-like behavior of artifacts. %2Acta Biotheoretica%1,%3XI%2,parts 3/4,
147-156
%7This review is filed as WEIZEN.REV[PUB,JMC] at SU-AI on the ARPA net.
Any comments sent to JMC@SU-AI will be stored in the directory PUB,JMC
also known as %2McCarthy's Electric Magazine%7. The comment
files will be designated WEIZEN.1, WEIZEN.2, etc.%1
.begin verbatim
John McCarthy
Artificial Intelligence Laboratory
Stanford, California 94305
.end
.once indent 0;
{date}